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Exact confidence intervals for the average causal effect 

on a binary outcome 

Xinran Li * and Peng Ding ^ 


Abstract 


Based on the physical randomization of completely randomized experiments. 


Rigdon and Hudgens 


(120151 ) propose two approaches to obtaining exact confidence intervals for the average causal 


effect on a binary outcome. They construct the first confidence interval by combining, with the 
Bonferroni adjustment, the prediction sets for treatment effects among treatment and control 
groups, and the second one by inverting a series of randomization tests. With sample size n, 
their second approach requires performing 0(n 4 ) randomization tests. We demonstrate that the 
physical randomization also justifies other ways to constructing exact confidence intervals that 
are more computationally efficient. By exploiting recent advances in hypergeometric confidence 
intervals and the stochastic order information of randomization tests, we propose approaches 
that either do not need to invoke Monte Carlo, or require performing at most 0(n 2 ) random¬ 
ization tests. We provide technical details and R code in the Supplementary Material. 
Keywords: Causal inference; completely randomized experiment; potential outcome; random¬ 
ization test; two by two table 


1 Notation and framework 


We extend the notation in 


Rigdon and Hudgensl (|2015T ). In a completely randomized experiment 


with n units, let Zj and Yj denote the binary treatment assignment and the binary outcome for unit 
j. We define yj( 1) and yj( 0) as the potential outcomes of unit j un der treat ment and control, and let 


N ik = : Vj{ 1 ) = h Vj{ 0) = k} for i,k = 0,1. Following 


Di ng an d Dasg uptal (1201 5). the potential 
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table N = {Nh,N\q, IVoi, Aoo) summarizes the potential outcomes for all units. The total numbers 
of units with treatment and control potential outcomes being one are N\ + = Y^j=iVj(X) an d 
N + \ = yj{ 0), respectively. The individual causal effect for unit j is 5j = yj(l) — yj(0), and the 

average causal effect is t(N) = ]C" =1 $j/ n = (Ai+ — N + i)/n = (Aho — No\)/n. Here we emphasize 
that t(N) is a function of N, and later we write it as r for simplicity. Let Z = [Z\ , Z 2 , ■ ■ ■, Z n ) be 
the treatment assignment vector, and the treated group is a simple random sample of size m from the 
n experimental units. The observed outcome of unit j is Yj = Z J yj(l) + (1 — Zj)yj(Q), a deterministic 
function of Zj and the potential outcomes (yj(l), yj (0)). We can summarize the observed data by 
four counts n zy = : Zj = z,Yj = y} for z, y = 0,1, and call n = (mi, nio, noi, noo) the observed 

table. The intuitive estimator, t = nu/m — noi/(n — m), is unbiased for r. 

Before observing the data, the potential table N can take any values as long as the sum of the 
N^’s is n. After obtaining n, the data put some restrictions on the potential table. A potential table 
N is compatible with the observed table n, if there exist potential outcomes {(z/j(1), 2/j(0))}” =1 , 
summarized by N, that give the observed table n under the treatment assignment Z. 

Theorem 1. A potential table N is compatible with the observed table n if and only if 


max{0, ran - N 10 , N u - n 01 ,N +1 - mo - mi} < mm{N n ,nn,N + i - n 01 ,n- N 10 - n 0 i - mo}- 


Theorem [T] gives an easy-to-check condition, which plays an important role in our later discus¬ 
sion. For all potential tables compatible with the observed table^_their r values mu st be equal to 


some k/n, with integer k between —(mo + n oi) an d nu + ?iqo (IRigdon and Hudgensl 2015). 


2 Confidence intervals without Monte Carlo 


We propose two approaches to constructing confidence intervals for r based on the hyper geometric 
distribution, which avoid Monte Carlo and are easy to compute. Let X ~ HyperGeo(A, T, S ) 
denote the hypergeometric distribution representing the number of units having some attribute in 


a simple r andom samp 


Recently, 


e of size S, which are drawn from T units with A units having this attribute. 


Wang 12015) improves classical hypergeometric confidence intervals, and proposes an 


optimal procedure to construct a confidence interval for A based on (T, S, X). Our discussion 
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below relies on this confidence interval for A based on a HyperGeo(H, T, S ) random variable X. 

2.1 Combining confidence intervals for iV 1+ and N + 1 

We can construct an exact confidence interval for r = (N±+ — iV_)_ i) / 77 ., by combining confidence 
intervals for IV 1 + and AT+i with the Bonferroni adjustment. Because the treated and control units 
are simple random samples of the n units in a completely randomized experiment, we have 


nn ~ HyperGeo (Ai+, n, m ), noi ~ HyperGeo (JV+i, n, n — m). 


We first obtain (1 — a/2) confidence intervals, [N± + , and [IV^, IV+-J, for N± + and N + 1 , and 
then use [(iV^_ — A^j/n, (N[ r + — N+^/n] as a (1 — a) confidence interval for r. 


2.2 A test statistic with simple null distributions 


We can construct exact confidence intervals for r by inverting a series o f randomization tests. 


However, the null distributions of \t — t\ is complex ([Rigdon and Hudgens 


2015). If we use the 


difference between the average causal effect on the treated units and r as the test statistic, then 
the null distribution has a simple form. The test statistic 


1 

m 


n 


J2 z jSj-r = 

3 =1 


n 1 AT 

—7-r^oi-A+i 

m\n — m) m 


( 1 ) 


is equivalent to noi ~ HyperGeo(fV + i, n, n — m), because (P) is a monotone function of noi, its only 
random component. Because the null distribution of noi depends only on N + 1 , potential tables 
with the same value of N +1 will yield the same p-value under randomization tests. Therefore, we 
need only to perform 0(n ) randomization tests according to all possible values of N + 1 between n 0 i 
and n — noo- The final lower and upper confidence limits for r are the minimum and maximum 
values of t(N) subject to (a) N is compatible with n, and (b) N yields a p-value larger than or 
equal to a. Constraint (b) is equivalent to restricting N+\ within a (1 — a) confidence interval 
[A r f J ], Ay7j], which helps avoid randomization tests or Monte Carlo. 
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3 Two-sided confidence intervals with fewer randomization tests 


We consider two sided confidence intervals for r using |fi — r| as the test statistic. We define 
P 2 (N) = Pn{\t — t| > |f obs — t|) with f obs being the realized value of f, which is the p -value of 
potential table N, or equivalently a null hypothesis, that is compatible with the observed table. 
We need to find all potential tables with P 2 > a, then use Theorem [T] to find the compatible ones 
among them, and eventually find the maximum and minimum r values. Without loss of generality, 
we assume m < n/2; otherwise we can switch the labels of Z. We consider the potential tables 
with r < r obs and t > f obs separately. We first focus on the potential tables with t < f obs , and 
rule out the ones with p 2 < a. 

For given Wi and JVoi, we define N_ 10 (Ni i, Wn) as the minimum value of Wo such that P 2 (N) > 
a and t(N) < fi obs , with N = (Wi) N w , Wi> n — Wi — Wo — Wi) being a potential table. If 
there is no such Wo> then we define Wo(Wi, Wi) = |_-^oi + nf obs J + 1, the smallest value of Wo 
such that t(N ) > f obs , where is the largest integer less than or equal to x. 

Theorem 2. (1) If N m < N' m , then Wio(^n^oi) < N 10 (N n , N^). 

(2) In balanced experiments with m = n/2, consider a potential table N with t(N) < f obs . 

P‘ 2 (N ) > a if and only if Wo > Wio(Wi, JVm). 

The computation burden arises because we need to perform randomization tests for all poten¬ 
tial tables compatible with the observed table. Fortunately, Theorem [2f 1) provides useful order 
information to reduce the number of randomization tests. We first assume that Wi is fixed. 
When W)i = 0, we find Wio(-^ii>0) by performing randomization tests starting from Wo = 0. 
When Wi increases to 1, we find IV 10 (Wi>l) by performing randomization tests starting from 
Wo = N.io(Nu,0) according to Theorem [2^1). Sequentially, when Wn increases by 1, we find 
W n(Wi, Wn) by performing randomization tests starting from Wo = Wo(-^n>-^oi — !)■ We re¬ 
peat this process until Wo increases to n — Wi- For a fixed Wi, we need to perform at most 0 (n) 
randomization tests. We implement the above procedure with Wi increasing from 0 to nn + noi, 
which requires at most 0(n 2 ) randomization tests in total. 

As long as we find N_io(Nn, Wu) for all possible (Wi,Wu)> we accept the potential tables 
compatible with the observed table such that Wo > Wio(-Wi> Wi) an d t(N) < f obs , with N = 


4 


(Nn, Nio, Nqi, n — Nn — Wo — Wi)- By switching the labels of Y, we can similarly accept some 
potential tables with r > f obs . The final lower and upper confidence limits for t are the minimum 
and maximum t values of accepted potential tables. 


The precedin g co nfi dence interval might 
ond approach in 


ae slightly wider than the interval obtained by the sec- 
Rigdon and Hudgensl (|2015l h because potential tables with W n > Wn(Wi, Nm ), 


t(N) < f obs might not satisfy P 2 > <x However, this interval will be the same as 


Rigdon and Hudgens 


(120151 ) in balanced experiments according to Theorem [2)^2). Our numerical and extensive simu¬ 
lation studies demonstrate that the above confidence interval coincides with the second one in 


Rigdon and Hudgens (2015) even though the experiments are extremely unbalanced. 


In balanced experiments, for every r within the above confidence interval, there exists a potential 
table N compatible with the observed table such that r(AT) = r and p 2 (N) > a. For general 
experiments, we can use the same idea to construct an exact one-sided confidence interval with 
0(n 2 ) randomization tests. We comment on these two issues in the Supplementary Material. 


4 Numerical examples 


We compare the confidence intervals for nr obtained by various procedures. Table [T] shows the 
observed tables and res ults. In our examples, the confidence intervals in Section [3] are the same as 


the second approach in 


Rigdon and Hudgensl (120151 ). even though some of them are very unbalanced. 


We have conducted extensive simulations for all observed tables with n = 24, finding that these 
two methods give the same 90%, 95% and 99% confidence intervals. 

Table 1: 95% confidence intervals for nr. “2.1”, “2.2” and “3” den ote the methods in Sections 12.11 
12.21 and l3l I and II denote the first and second approaches in Rigdon and Hudgensl ( 2015 ): “#(3)” 
and “^(11)” denote the numbers of randomization tests needed for “3” and “II.” 


n 

2.1 

2.2 

3 

I 

II 

#(3) 

#(H) 

(1,1,1,13) 

[-2,14] 

[-1,14] 

[-1,14] 

[-2,14] 

[-1,14] 

103 

112 

(2, 6,8,0) 

[-14,-3] 

[-14,-2] 

[-14,-5] 

[-14,-2] 

[-14,-5] 

113 

189 

(6,0,11,3) 

[-5,8] 

[-11,7] 

[-4, 8] 

[-5,7] 

[-4,8] 

283 

336 

(6,4,4, 6) 

[-6,12] 

[-6,11] 

[-4,10] 

[-6,12] 

[-4,10] 

308 

1225 

(1,1,3,19) 

[-4,20] 

[-3,20] 

[-3,20] 

[-4,20] 

[-3,20] 

251 

320 

(8,4, 5, 7) 

[-4,14] 

[-6,14] 

[-3,13] 

[-6,15] 

[-3,13] 

421 

2160 
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Supplementary material 


Section IA.1I describes a procedure to obtain a one-sided confidence interval using f as the test 
statistic, which requires 0(n 2 ) randomization tests. Section [A.21 contains proofs of the theorems. 
Section IA.3I comments on some computational details. In the following, we use a V b = max(a, b ) 
and a A b = min(a, b). 

A.l One-sided confidence interval 

Without loss of generality, we consider the lower confidence limit for r using f as the test statistic 
in randomization tests. We define pi(N ) = Pn(t > f obs ) with r obs being the realized value of t, 
which is the p-value for potential table N compatible with observed table. To obtain the (1 — a) 
lower confidence limit, we need to find all potential tables with pi > a, then use Theorem |T] to find 
the compatible ones among them, and eventually find the minimum r value. 

To facilitate computation, it is crucial to exploit order information of the potential tables. For 
given Nn and IVoi, we define N w (Nu, Nm) as the minimum value of Ni o such that pi(N) > a 
with N = {Nu,Nio,N 0 i,n — Nu — Ni o — AT 0 i) being a potential table. If there is no such Ni o, 
then we define N-, n(Ni, Nqi) = n + 1. 

Theorem A.l. (1) If N 0 i < N^, then N 10 (Nu,N 0 i) < N 10 (Nu, N^). 

(2) For a potential table N, pi(N) > a if and only if Ni o > jV 10 (AIn, Nm)- 

Theorem IA.1I provides useful order information to reduce the number of randomization tests, 
because we do not need to test for all potential tables. According to Theorem IA.ll 2j. we need only 
to find Aio(IVii,IVoi) for given (Nu, Nqi). We first fix Nu. When A r oi = 0, we find A 10 (IVn , 0) 
by performing randomization tests starting from Ni o = 0. When Nqi increases to 1, we find 
N_io(Nu, 1) by performing randomization tests starting from Ni o = iV 10 (Ain, 0) , because Theorem 
lA.lf l) guarantees that N_ w (NuA) > Aio(A^n,0). We repeat this process until Alio increases to 
n — An. For a fixed An, we need to perform at most 0(n ) randomization tests. We implement 
the above procedure with An increasing from 0 to nu + ^oi 5 which requires at most 0(n 2 ) ran¬ 
domization tests in total. For every r within the final confidence interval [/, (nu + ^oo )/ n ], there 
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exists a potential table N compatible with the observed table such that r(iV) = r and pi(N) > a. 


We comment on the computational details in Section I A. 31 and show some numerical examples in 
Tabic [Adi 


Table A.l: 95% one-sided confidence interval 


n 

Cl 

n 

Cl 

(1,1,1,13) 

[-1,14] 

(2, 6,8,0) 

[-14,2] 

(6,0,11,3) 

[-3,9] 

(6,4,4, 6) 

[-3,12] 

(1,1,3,19) 

[-3,20] 

(8,4, 5, 7) 

[-2,15] 


A.2 Proof of the theorems 

Proof of Theorem Q} Let Xjfc denote the number of units in set {j : yj( 1) = i,2/j(0) = k} that are 
assigned to treatment. We have 

xu + afio + ^oi + %oo = x 11 + x 10 = nn, (An - x u ) + (N 01 - x 0 i) = n 01 , 

0 < x ik < N ik (i, k = 0,1). 

The potential table N is compatible with the observed table n if and only if the above equations 
have integer solutions for (in,xio, xoi, £oo)- ^ ie a bove equations are equivalent to 

x w = n 11 -xn , x 0 i = Nn +N 0 i - n 0 i - xn, x 00 = x n + n 0 i + ni 0 - N u - N 01 , 

0 <x ik <N ik {i,k = 0,1). 

The integer solutions exist if and only if there exists an integer in satisfying 

0 < x u < An, 0 < nu — xu < Nw, 

0 < An + Am — noi — xu < A 0 i, 0 < in + n-oi + nio — An - A^^qi < A 0 o, 
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which are equivalent to 


max{0, nn - N 10l N u - n 0 i,N u + N 01 - n 10 - n 0 1} 

< *11 < min{ATn, nn, iVn + N 01 - n 0 i, N 00 + N u + N 01 - n 0 i - ni 0 }. 

Therefore, we have proved Theorem [TJ □ 

In order to prove the theorems, we need to introduce additional notation and lemmas. Define 

r = {(0,1,0, -1), (-1,1,0,0), (1,0, —1,0), (0,0, -1,1)}, 
r0 = {(-1,1,0,0), (0,0,-1,1)} c r. 

If two potential tables of sample size n, N and N', satisfy N' = N + A with A € T, then 
t(N') = t(N) + 1/n. 

Lemma A.l. If N' = N+ A with A € 1~, then we can construct potential outcomes {x/j (1), 2/y (0)}^ =1 
and {y'(l),y'(0)}” =1 such that only one unit r is different, i.e., (y r (l), y r (0)) = (ai,a 2 ), (y' r (l),y' r (0)) = 
(61,62)1 and (yj(l),yj(0)) = (y'j (1),y'(0)) for all j ^ r. We show the corresponding values of A, 
(ai,a 2 ) and (61,6 2 ) in Table [A~2l 

Table A.2: The difference between two potential tables 


A 

(oi, 0,2) 

(61,62) 

(0,1,0,-1) 

(0,0) 

(1,0) 

(-1,1,0,0) 

(1,1) 

(1,0) 

(1,0,-1,0) 

(0,1) 

(1,1) 

(0,0,-1,1) 

(0,1) 

(0,0) 


Lemma A. 2 . Assume that N' = N + A with A € 7 ”, and N and N' differ by only one unit r 
as constructed in Lemma I A. 11 Let 5 be the random indicator for the rth unit being assigned to 
treatment. The relationship between f' — f and A is shown in Table IA31 

Proof of Lemma \A.“A Let u be an outcome of the sample space, and treatment assignment Z(oj) 
is a function of uj. For each assignment Z(u), let Xik(co) denote the number of units in set {j : 
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Table A.3: Difference between t' and t 


A 

f — f 

(0,1,0,-1) 
(-1,1,0,0) 
(1,0,-1,0) 
(0,0,-1,1) 

5/m 

(1 — 5)/{n — m) 
5/m 

(1 — 5)/{n — m) 


(yj(l).yj(O)) = ( i,k )} that are assigned treatment, and x' ik (u ;) the number of units in set {j : 
(y'-(l), y'j{0)) = ( i,k )} that are assigned treatment, i,k = 0,1. Then we have 


r(w) =—(xn(w) + xio(w))--— {(An - xn(w)) + (A 0 i - ®oi(w))} , 

m. n — m 

r\u) =— (®' n (u;) + xi 0 (w))-— {{Ki - ®'u(w)) + ( N m ~ *oi( w ))} • 

m n — m 


Because An, Aq 15 x' ik (u) are functions of An, Aoi, Xifc, <5(w), as shown in Table IA~H we can 
immediately obtain t(u) — t'(u) as shown in Table [/Oil 

Table A.4: Quantities of N' as functions of quantities of N 


A 

*ii(w) 

x' 10 (w) 

x' 0 i(w) 

N[ i 

Ki 

r(IV') 

(0,1, 0,-1) 

Xu (w) 

zioM + <5(w) 

x 0 i(w) 

An 

N 0 i 

t(N) + 1/n 

(-1,1,0,0) 

Xn(w) - <5(w) 

zioM + 5(uj) 

x 0 i(w) 

An-1 

IVqi 

t(N) + 1/n 

(1,0,-1,0) 

xn(w) + <5(w) 

xio(w) 

.t 0 i(o;) - <5(w) 

An +1 

A 0 i — 1 

r(IV) + 1/n 

(0,0,-1,1) 

x n (w) 

xio(w) 

zoiM - 5(uj) 

An 

A 0 i — 1 

t(N) + 1/n 


□ 

To prove Theorem [2l we need the following lemmas. 

Lemma A.3. If m < n/2, assume N' = N + A with A € To- If t(1V) < f obs and t(N') < f obs , 
then P 2 (N') > p 2 (N). 

Proof of Lemma \A. 31 We consider two potential tables N and N' constructed in Lemma IA.21 For 
any treatment assignment Z(ui) and A £ T, we have f(cj) < < f(uj) + l/(n—m) < f{u)+2/n, 

and 

2 t — t(lo) = 2r H-> 2 t H- t(uj) — — = 2t — fiuj). 

n n n 
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If r < f obs and t' < f obs , then 

p 2 {N') =P N '(\t' - t'\ > f obs - t') = P N ,{ max{r', 2 r' - t'} > f obs ) 

>P/v(max{f, 2 t — f} > f obs ) = Pn( |t — t| > f obs — r) = p 2 (N). 


Here we change the probability measures from P N / to Pj\r because of the coupling in Lemma lA.il 
and Lemma I A . 21 i.e., 


P n ,(t' = 


) = P) S {tmix'r'lu), 2t' - f'(w)} > T°>"} 

' ' f, J 


> ^ 'j l{max{f (cj), 2r — f(w)} > f obs } = Pn(t > 


iobs\ 


□ 


Lemma A.4. If m < n/2, assume IV and N' are two potential tables. If t(N ) < f obs , t(N') < 
r ohs , and NP = Nn, N[ 0 = < N m , then p 2 (N') > p 2 (N). 


Proof of Lemma A. f\ Because r(N) < f obs , r(IV / ) < r obs , and 


AT + (Aoi -N'i) • (0,0, -1,1) = (N[i,N[ 0l N'i,N' 0 ) = N', 


we know p2(N') > p 2 (N ) according to Lemma I A. 3 1 □ 

Lemma A. 5 . In balanced experiments, assume IV , = N + A with A € T. If t(IV) < f obs and 
t(N') < f obs , then p 2 {N') > p 2 (N). 

Proof of Lemma I A., 51 We consider two potential tables IV and IV' constructed in Lemma IA.2I For 
any treatment assignment Z(u) and A £ 7 ~, we have t(u) < f'(u) < f (w) + 1/m, and 

2 2 1 

2 t — = 2r H-f'(u;) > 2r H- t (w)-= 2r — f (w). 

n n m 
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If r < f obs and t ' < f obs , then 

p 2 (N') =P N '(\t' - t'\ > f obs - t') = P N ,{ max{f', 2 r' - t'} > f obs ) 

>P/v(max{f, 2 t — f} > f obs ) = Pn( |t — t| > f obs — r) = p 2 (N). 


□ 

Lemma A.6. In balanced experiments, assume IV and IV 7 are two potential tables. If t(IV) < 
f obs , t(N') < f obs , and = N n ,N[ 0 > N W ,N^ < N 01 , then p 2 (N') > p 2 (N). 

Proof of Lemma \A.(A Because r(IV) < f obs , t(N') < f obs , and 

N + (IVoi - A' x ) ■ (0,0, -1,1) + (A' 0 - N 10 ) • (0,1,0, -1) = (A',, A' 0 , A' l5 A' 0 ) = N 

we know p 2 (N') > p 2 (N) according to Lemma I A. 51 □ 

Proof of Theorem |3 We first prove (1). We show fV 10 (Aii, IVoi) < IV 10 (IVii, A^), for any 
0 < IVn < n and 0 < Aoi < Aq X < n — An. If IV 10 (IVii, N' m ) > |_Aoi + nf obs J + 1, then the 
conclusion holds trivially. If Ai 0 (An, N' Ql ) < [IVoi + nf obs J < + ?if obs J, then 

Ni = (An, K w (A n , N^N^n-Nu- K w (N u , A'i) - N ',) 

is a potential table satisfying r(IVi) < f obs and p 2 {Ni) > «• Because 

An + A 10 (An, A'i) + Aqi < An + A 10 (An, A' x ) + < n, 

we know that 


N 2 = (An, Ai 0 (A n , A'i), A 0 i,n - A n - Ai 0 (A n , A' x ) - A 0 i) 
is a potential table which satisfies 

t(N 2 ) = (Ai 0 (Aii, A'i) - A 0 i)/n < (LA 0 i + nf obs J - A 0 i)/n < f obs . 
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According to Lemma |XH p 2 (N 2 ) > p 2 {N i) > a , implying Wlo(Wi, Wn) < Wlo(Wi, Wu)- 

We then prove (2). For a potential table N with t(N ) < f obs , if Wo < Wio(W 1 , A r 0 i), then 
Pi(N) < a from the definition of Wlo(Wi> Wi)- Otherwise, W nfWi , Wi) < Wo £ |_Wi + nf obs J, 
which implies that 


N 3 = (A n , JVio(A n , N 01 ), N 01 , n - Wi - N w (Wi ,N 01 )~ N 01 ) 

is a potential table satisfying r(JV 3 ) < f obs and p 2 (N 3 ) > a. According to Lemma I A. 61 p 2 (N) > 
P2(N 3 ) > a. 

□ 

To prove Theorem lA.il we need the following lemmas. 

Lemma A.7. If N' = IV + A with A eT, then pi(N') > p 3 (N). 

Proof of Lemma \A. 7| We consider the potential tables constructed in Lemma IA.11 For any treat¬ 
ment assignment Z(uj), we have t'(cu) > f( oj), and therefore pi(N') = P N ff > f obs ) > Pn(t > 
f obs ) = pi(N). □ 

Lemma A.8. If N' and N satisfy A[ 1 = Nn,N [ 0 > Wo and Wn — Wij then pi(N') > pi(N). 

Proof of Lemma 1 .4. Al Because 

N + (W)i - iV'i) ■ (0,0, -1,1) + (N[ o - Wo) • (0,1, 0, -1) = (N^, N[ 0 , A' l5 A' 0 ) = AT', 

we have pi(N') > pi(N) by repeatedly applying Lemma [A. 71 □ 

Proof of Theorem IA.ll (1) If Wio(Wij Wi) = n +l, the conclusion holds trivially. If Wio(Wii Wi) — 
n, then 

Ni = (Wi, Wo(Wi Wol) ,N^,n - Wi -N w (Wi, AT'i) - A' x ) 
is a potential table and satisfies pi(N\) > a. Because Wi < IVqi> 

N 2 = (Wi, Wo (Nn , iV'i), N 01 , n - N n - W10(Wi, Wi) - Wi) 
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is also a potential table. Lemma lA. 81 further implies Pi (JV 2 ) >Pi(N\) > a. Therefore A 10 (An, Aoi) < 
N 10 (Nu,N^). 

(2) For a potential table N, if Aio < A 10 (A r n. Aoi), then pi(N) < a by the definition of 
N U) (Nn, A r oi). Otherwise, A 10 (An, Aqi) < Aio < n + 1, implying that 

N 3 = (An , N 10 (Nu, Noi),N 0 i,n-Nn- jV 10 (N n ,N m )- A 0 i) 

is a potential table and satisfies pi(N^) > a. According to Lemma fA.81 pi(N) > pi(N^) > a. 

□ 


A.3 More computational details 

Theorem A. 2 . The procedure for the two-sided confidence interval in Section [3] requires at most 
0(n 2 ) randomization tests. 

Proof of Theorem \A/A For any fixed An, it must be true that A 10 (An, Aoi) = [Aoi + nf obs J + 1 
for 0 < N m < 0 V [—nf obs ] without doing any randomization tests, where [x] denotes the smallest 
integer greater than or equal to x. Hence, in order to get A 10 (An, Aoi) for all 0 < Aoi < n — An, 
the number of randomization tests needed is less than or equal to 

n—Nn 

n AA 10 (An,0V [-nf obs ]) + 1 + ^2 {nAA 10 (An,fc) —nAA 10 (An,fe —1) + 1} < 2n + l, 

fc=0V[-nf obs l+l 

where n A A 10 (An, 0 V [—nf obs ]) + 1 bounds the number of randomization tests needed for Aoi = 
0 V [—nf obs ], and n A A 10 (An, k) — n A A 10 (An, k — 1) + 1 bounds the number of randomization 
tests needed for Aoi = k (0 V |~—nf obs ] + 1 < k < n — An). 

Because the number of possible values of An is less than (n+1), the total number of randomiza¬ 
tion tests needed for calculating the lower confidence limit is less than or equal to ( 2 n + l)(n + 1 ) = 
0(n 2 ). The computation for the upper limit of t is the same as the lower limit by switching the 
labels of Y. Therefore, the total number of randomization tests needed is at most 0(n 2 ). □ 


Theorem A.3. The procedure for the one-sided confidence interval in Section [Aj] requires at most 
0(n 2 ) randomization tests. 










Proof of Theorem \A.3\ . For any fixed IVn, in order to get A 10 (Aii, A r 0 i) for all 0 < Aoi < n — An, 
the number of randomization test needed is less than or equal to 

n—TVn 

n A Ain(Aii, 0) + 1 + ''f ^ {n A Aiq(A n, k ) — n A A-i n( Ai i , k — 1) + 1} ^ 2n + 1, 

k =1 

where n A A 10 (An,0) + 1 bounds the number of randomization tests needed for Aoi = 0, and 
n A A 10 (Ahi, k) — n A iV 10 (A^n, k — 1) + 1 bounds the number of randomization tests needed for 
Nqi = k{l<k<n — N n ). 

Because the number of possible values of Nn is less than (n + 1), the total number of random¬ 
ization tests needed is less than or equal to ( 2 n + l)(n + 1 ) = 0 (n 2 ). □ 

Mathematically, by inverting a series of randomization tests, we obtain confidence sets for r. 
These confidence sets in Sections 151 and [~Y~T1 mav not be intervals. The final theorems ride out this 
possibility, and confirm that these confidence sets are indeed confidence intervals. In order to prove 
the final two theorems, we need to introduce the following lemma. 

Lemma A.9. For any potential table N compatible with the observed table n, if t(N) < (nn + 
n 00 )/n, then there exist a potential table N' such that N is compatible with the observed table 
and N' = N + A with A G T. 

Proof of Lemma \A.fA Because N is compatible with the observed table n, there exist potential 
outcomes {j/y (1), yj(0)}” =1 , summarized by IV, that give the observed table n under the treatment 
assignment Z. We construct potential outcomes {y'(l), y'-(0 )}” =1 different from {jjj(l), yj(A)}]=\ 
by only one unit r, i.e. Z r = z, (y r (l),y r (0)) = (ai,a 2 ), (y'(l), y' r (0)) = (&i,f> 2 ), and (yj(l),yj(0)) = 
(yf (1), yf (0)) for all j ^ r. We show the corresponding values of 2 ,( 01 , 02 ) and ( 61 , 62 ) in Table 
IA.51 where N' denotes the potential table summarizing (y'(l),yj(0)}" =1 , and A = IVi — N. 

Table A.5: Constructing potential table N' 


2 

(01,02) 

(61,62) 

<1 

II 

£ 

1 

0 

(0,0) 

(1,0) 

(0,1,0,-1) 

1 

(1,1) 

(1,0) 

(-1,1,0,0) 

0 

(0,1) 

(1,1) 

(1,0,-1,0) 

1 

(0,1) 

(0,0) 

(0,0,-1,1) 
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Because potential outcomes {y'-(l),y'-(0)}” =1 give the same observed table n under the treat¬ 
ment assignment Z, N' is compatible with the observed table. We need only to show that unit r 
exists if t(N) < (nn + noo)/n. If such unit r does not exist, then the following must be true: 

iVoo — •'Coo = 0, xn = 0, A r 0 i — xoi = 0, xqi = 0, (A.l) 


recalling that xik denotes the number of units in set {j : yj( 1 ) = i,yj( 0 ) = k} that are assigned to 
treatment under the treatment assignment Z. Formula (IA.1I) implies N = (noi,nn + noo, 0, nio) 
and t(7V) = (nn + noo)/n, which contradicts r(IV) < (nn +noo)/n. Therefore, (1A.1D cannot hold 
and the unit r must exist, and Lemma I A. 9 1 holds. □ 


Theorem A. 4. In balanced experiments, the final two-sided confidence set in Section [3] must have 
the form [l,u] for some values l and u, in the sense that for every possible r e [Z,n], there exists a 
potential table N compatible with observed table that satisfies r(N) = r and P 2 (N) > a. 


Proof of Theorem A. 41 For any r < f ohs , if there exists a potential table N satisfying t(N) = r 
and p 2 (N) > a , then according to Lemma I A. 91 and Lemma lA.51 there exists a potential table N' 
satisfying r(N') = t + 1/n < f obs and p 2 {N') > a. Similarly, by changing the labels of Y, we 
know that for any r > r obs , if there exists a potential table N satisfying r(IV) = r and p 2 (N) > a, 
then there exists a potential table N" satisfying t(N") = t — 1/n and p 2 (N") > a. Therefore, 
Theorem IA.4I holds. □ 


Theorem A. 5 . The final confidence set in Section lA.ll must have the form [l, (nn + noo)/n] for 
some value l, in the sense that for every possible r > l, there exists a potential table N compatible 
with observed table that satisfies t(N) = t and pi(N) > a. 

Proof of Theorem \A.5l For any possible r < (nn + noo)Ah if there exists a potential table N 
satisfying t(N ) = r and p\(N) > a , then according to Lemma [A. 91 and Lemma lA.71 there exists a 
potential table N' satisfying t(N') = r + 1/n and pi(N') > a. Therefore, Theorem I A. 51 holds. □ 
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